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ABSTRACT 

This paper presents the online cognitive and algorithm tests, which were developed in order to determine if certain 
cognitive factors and fundamental algorithms correlate with the performance of students in their introductory computer 
science course. The tests were implemented among Management Information Systems majors from the Philippines and 
among the students of the Information Systems track in a technology high school in Japan. The cognitive factor. Spatial 
Scanning, correlated significantly with the performance of both student groups in introductory computer science. Among 
the algorithms, Maze Tracing correlated with the university students’ final grade and Binary Search with the midterm 
exam result of the high school students. The online tests also showed significant correlations among Searching, Sorting 
and Maze Tracing algorithms. The obtained correlations may be considered in creating assessment tools and in designing 
learning strategies for introductory computer science courses for both university and high school levels. 
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1. INTRODUCTION 

Researchers, particularly in the field of Computer Science Education, have endeavored to find ways on how 
to address the high drop-out rate and the poor performance of students in computing. These studies (Irons D., 
1982, Rountree, N., et.al., 2004, Simon, et.al, 2006) attempt to identify the factors that determine success in 
computing and measure aptitude in programming. In relation to this, the importance of mastering 
programming early was emphasized by the ACM Computing Curricula 2001 Final Report which proposes 
programming as a prerequisite for many advanced courses in computer science (ACM, 2001). It is also 
important to find ways to improve computer science for high school as it is now being considered as an 
essential subject along with the traditional STEM (science, technology, engineering and mathematics) (CSTA, 
2005). 

The main goal of this research is to determine which among known cognitive factors and fundamental 
algorithms correlate with the actual student performance in introductory computer science through online 
tests. The tests can be used as reference for designing assessment tools for both university and high school 
students who plan to study computer science and other technology and computing-related degrees. The 
correlation results may also be used in determining possible ways to help students succeed in introductory 
computer science, specifically in programming. 


2. METHODOLOGY 

The first phase of this research covered the initial development of the online cognitive and algorithm tests 
and their preliminary implementation among sixty-one (61) second year Management Information Systems 
(MIS) majors at the Ateneo de Manila University. These participants have taken up Introduction to 
Computing I (CS 21A) and were enrolled in Data Structures and Algorithms (CS 21B) when they took the 
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tests in May 2007. The second phase involved the translation of test instructions to Japanese and their 
implementation in September, 2011 among forty (40) students from the Information Systems track of 
TokyoTech High School. At the time of the implementation of the tests, these students have already finished 
introductory programming but have not yet studied the fundamental algorithms considered in this research. 

2.1 The Cognitive Factors Tests 

Two sets of online tests were developed for the purposes of this research. One set was based on known 
cognitive factors from the Kit of Factor-Referenced Cognitive Tests by the Educational Testing Services 
(ETS), a US-based educational institution. Each cognitive factor has corresponding paper tests. The first 
cognitive test, Object-Number for the cognitive factor Associative Memory, assesses the ability of the student 
to learn combinations of words and numbers. The second is Letter Sets Test for Induction, which aims to find 
the rule that makes the letter sets similar and to identity which set does not fit the rule. The third. 
Diagramming Relationships test for Logical Reasoning, entails finding the relationships among things in a 
group, represented by overlapping circles. The last is Map Planning test for Spatial Scanning, the goal of 
which is to find the shortest route between two points in a city map the shortest possible time and according 
to a set number of rules (Ekstrom, et.al, 1976). A sample map for the Map Planning Test is shown in Figure 1 
below. Previous researches have shown that the aforementioned factors are related to programming aptitude 
(Irons, 1982; Scanlan, 1988). For the Japanese high school students. Diagramming Relationships Test was 
not included because this particular test uses English words and the authors have no license to modify and 
translate actual exam contents. 
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Sample routes 
A to Z passes building 1 
E to S passes building 5 
P to J passes building 3 


Figure 1. Sample map and routes in Map Planning Test 


2.2 The Algorithm Tests 


Fundamental algorithms, commonly taught in introductory computer science were designed as games that 
can simulate how the algorithms work. The first set of games is Card Searching which includes Linear 
Search and Binary Search. The goal is to locate a certain number from among ten (10) numbered cards. The 
next is Sorting games, where the goal is to simulate Insertion Sort and Selection Sort while sorting ten 
numbered cards in ascending order. The third game is Maze Tracing which attempts to simulate how breadth- 
first and depth-first search algorithms work. The last algorithm. Tower of Hanoi, checks the student's ability 
to think recursively. It was not implemented among the high school students because recursion is not covered 
in their course syllabus. Figure 2 shows the screen components of Card Sorting. 
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CLICK TO CHOOSE GAME 

( FIRST GAME SECOND GAME 

Number ofTries 

2 tries game 1 

Number of Cards Chosen 

0 

Number of Moves 

0 

Number of Times each card is Moved 

0000000000 

TIMER 

CLICK HERE TO SAVE GAME STATS 

0:3 


Figure 2. Unsorted cards and data boxes in Card Sorting 


246 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


3. DATA ANALYSIS 

The scores from the online cognitive factors tests were collected. For the algorithm tests, data based on the 
execution of the games were used to compute the test scores. The test results were then correlated with the 
actual performance of the students in their introductory computer science course, the grades in CS 21A of the 
university students and the scores in the mid-term exam on algorithms of the high school students. Data 
analysis was done using Pearson's R correlation function in SPSS. 

3.1 Correlations of Online Tests with Performance in Introductory Computer 
Science 

Among the cognitive factors. Spatial Scanning correlated with the performance of the university students in 
CS21A (r=.331, R=.1095, sig=.010). This finding is similar to the study of Irons where this factor is one of 
those considered to be most relevant to programming aptitude. Spatial Scanning is associated with debugging 
skills as it entails finding “particular configurations representing a pathway through the field”. This is 
considered as a more psychologically relevant factor in predicting programming skill and is greatly related 
with debugging ability (Irons, 1982). Spatial Scanning also correlated with the midterm exam results of the 
high school students specifically with their score in the Binary Search problem (1-.383, R=.1466, sig=.015). 
This can be related with the ability to apply Binary Search algorithm where a certain scheme is employed for 
locating values in an array. Among the algorithms, only Maze Tracing correlated significantly with the 
performance of the university students in their introductory computer science class, CS21A (1-.310, R=.0961, 
sig=.017). In the case of the high school students, the result of the online test on Binary Search correlated 
significantly with the score in the Binary Search problem in the midterm exam (r=.352, R=.1239=, sig=.026). 
This particular result implies that it is more substantial to find the correlations between the online tests with 
the performance in the actual application of the algorithms rather than with the final grade. 

3.2 Correlation Results among Algorithms 

The results of the online algorithm tests were also correlated against each other as shown in Table 1. Two 
significant correlation results are similar for both university and high school participants. Linear Search with 
Binary Search and Insertion Sort with Selection Sort. Linear Search and Binary Search tests are expected to 
correlate because both are search algorithms and have the same objective, that of locating a value from a 
given set of cards. The same applies to the correlation between Insertion Sort and Selection Sort. 


Table 1. Correlation results among the algorithm tests 


Algorithms 

Correlation values ! 

University students 

High school students 

Linear Search with Binary Search 

1-.555, R=.3080, sig=.000 

r=.859 R=.7378, sig=.000 

Insertion Sort with Selection Sort 

1-.517, R=,2672, sig=.002 

r=,511, R=.2611,sig=.001 

Binary Search with Maze Tracing 

1--.314, R=.0985, sig=.023 

r=.465, R=. 2162,sig=.002 

Binary Search with Insertion Sort 

1-.501, R=.2510, sig=.003 


Binary Search with Selection Sort 

r=.375, R=.1406, sig=.011 


Maze Tracing with Tower of Hanoi 

1-.345, R=. 1 190,sig=.007 


Maze Tracing with Linear Search 


r=.390, R=.1521, sig=.013 


There are a number of differences, though, between the test results of the university students and high 
school students. For the university students. Binary Search and Maze Tracing are inversely correlated. These 
students who already knew Binary Search could have implemented the algorithm’s strategy in the 
corresponding online test but tried free search in Maze Tracing. However, for the high school students. 
Binary Search and Maze Tracing have positive correlations. This may be explained by the fact that searching 
is naturally applied in maze traversal. Both processes entail some movement from one point to another. 
Another possible reason is that the high school students have not yet studied Binary Search at the time of the 
online test implementation. Thus, they could have applied free searching in both algorithm tests, that is, not 
following a learned method. The same reason may be applied with the positive correlation between Maze 
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Tracing and Linear Search. Binary Search further correlated with Insertion and Selection Sort and Maze 
Tracing with Tower of Hanoi for the online test results of the university students but not for the high 
school students. These other correlations may be again due to the fact that the university students have 
already studied algorithms by the time of online test implementation. Their approach in executing algorithms 
is more unified and structured. Generally, the correlations may also be due to the natural connection between 
the concerned algorithms. Search strategies are naturally employed when sorting or tracing a maze. 


4. CONCLUSION 

Online cognitive and algorithm tests were developed and implemented among university students who are 
majoring in information systems and high school students who are in a similar course track. The aim was to 
find out correlations among the known cognitive factors and fundamental algorithms with the performance of 
the students in their introductory computer science course. 

A number of significant correlations were obtained and similarities as well as differences in the 
correlations between the two groups of participants were observed. The correlation results indicate 
consistency in the students’ performance in the online test and demonstrate certain reliability among the 
developed algorithm tests. Moreover, the obtained results show close connections among the cognitive factor 
Spatial Scanning and Maze Tracing and Binary Search, making them possible factors to be considered in 
learning computing fundamentals. 

In summary, the online cognitive factors and algorithm tests that correlated with the performance of the 
students either with the final grade or exam score may then be considered in designing tools in assessing 
student ability to succeed in an introductory computer science course. The said cognitive factors and 
algorithms may also be taken into account in the development of a course outline for introductory computer 
science and in preparing actual problem exercises for the students. 

Future work for this research includes considering other cognitive factors and algorithms for the online 
tests. The other data on the online test performances that were not used in scoring the algorithm tests may 
also be used in constructing other formulas for scoring the tests. These data may then be used to further 
study particular aspects with which the students implement algorithms and determine other factors that can 
assess aptitude in introductory computer science courses. 
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